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METHOD AND APPARATUS FOR CORRECTING DIGITAL IMAGES 

10 Inventors 

Robert W. Turner 
Pauline Joe 
Mark A. Zenor 

Field of the Invention 

1 5 This invention relates generally to digital cameras and, more particularly, to electro- 

optic framing cameras. 

Related Applications 

This application is related to concurrently- filed U.S. Patent Application Ser. No. 

_, atty. docket number BOEI-1-1180; and U.S. Patent Application Ser. No. 

20 , atty. docket number BOEI-1-1181, all filed and all hereby 

incorporated by reference. 

Background of the Invention 
Homeland defense, environmental base-lining, emergency responses to natural and 
manmade disasters, agricultural analysis, and other change detection applications need an 
25 ability to acquire real-time, high-resolution spatial and multispectral imagery to support 
situational evaluation of an environment. In order to effectively and efficiently support 
situational evaluation, multispectral image generation must be accurate and attainable at a 
minimum cost. For homeland security and environmental monitoring, low cost multispectral 
sensors can be used to generate digital multispectral images. However, digital images 
30 generated by multispectral sensors, such as electro-optic framing cameras, may contain 
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image streaking caused by electronic noise of the sensor. The streaking results in a decrease 
in the radiometric accuracy of images. Therefore, removal of streaking effects is important 
for producing more radiometrically accurate images. 

Multispectral sensors or digital cameras with electronic shutters, such as a Phillips 
5 liquid crystal device or other cameras with frame store devices, cause light scattering that 
also reduces the radiometric accuracy of an image. 

Therefore, there exists an unmet need to produce low-cost, radiometrically accurate 

images. 

Summary of the Invention 

10 The present invention provides methods, systems, and computer-based program 

products for correcting image streaking, light scattering, and other effects in digital images 
generated by multispectral sensors. 

According to exemplary embodiments of the present invention, an image frame (i.e., 
the image frame is what a user desires to view), a plurality of image frames from the digital 

15 sensor when exposed to dark, i.e. light below a predefined luminance level, and a plurality of 
image frames from the digital sensor when exposed to light above a predefined luminance 
level are received. The received image frame is corrected based on the received plurality of 
image frames exposed to dark and the plurality of image frames exposed to light. 

In accordance with further aspects of the invention, each image frame includes a 

20 plurality of frame units. A mean value for each frame unit of the plurality of image frames 
exposed to dark and a mean value for each frame unit of the plurality of image frames 
exposed to light are calculated. A first image frame is generated by subtracting the calculated 
mean value for each frame unit of the plurality of image frames exposed to dark from the 
corresponding frame unit in the received image frame. A second image frame is generated by 

25 subtracting the calculated mean value for each frame unit of the plurality of image frames 
exposed to dark from the calculated mean value for each corresponding frame unit of the 
plurality of image frames exposed to light. A third image frame is generated by dividing each 
frame unit in the first image frame from the corresponding frame unit in the second image 
frame. An average of frame units is calculated within a predefined center section of the 

30 second image frame. A fourth image frame is generated by multiplying each frame unit in the 
third image frame by the calculated average of the center section frame units, thereby 
producing a corrected image frame of the received image frame. 

In accordance with other aspects of the present invention, all the received image 
frames are enhanced based on characteristics of the digital sensor. Enhancing a region of 

35 interest is determined based on evaluating a masked region of the digital sensor, which 
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prevents light from reaching the sensor detectors. A standard deviation of the frame units is 
determined for a determined region of interest within the masked region. A threshold value is 
determined based on the determined standard deviation. A mean of the frame units in the 
determined region of interest that are below the determined threshold value is determined. 
5 Enhanced image frames are generated based on the determined mean of the frame units and a 
coefficient. 

In accordance with still further aspects of the present invention, light scattering 
effects are removed from the corrected image frame based upon characteristics of the digital 
sensor. The characteristics of the digital sensor include light scattering effects of each frame 
10 unit upon each frame unit. Light scattering effects are based upon distance between the 
respective frame units. 

Brief Description of the Drawings 

The preferred and alternative embodiments of the present invention are described in 
detail below with reference to the following drawings. 
15 FIGURE 1 is a block diagram of an exemplary system formed in accordance with the 

present invention; 

FIGURES 2-5, and 8 are flow diagrams of an exemplary process performed by the 
system shown in FIGURE 1; 

FIGURE 6 illustrates aspects of the process shown in FIGURE 4; 
20 FIGURE 7 illustrates image graphics as it relates to a portion of the process of 

FIGURE 6; and 

FIGURE 9 is a graphical example of some of the processing performed in the process 
of FIGURE 8. 

Detailed Description of the Invention 

25 The present invention provides methods, systems, and computer-based products for 

correcting image streaking, light scattering, and other effects in digital images generated by 
multispectral sensors. 

FIGURE 1 illustrates an exemplary system 40 for producing radiometrically accurate 
images. The system 40 includes a sensor 44, a processor 46, memory 48, a display 50, and a 

30 user interface 52. The processor 46 is suitably coupled to the sensor 44, the memory 48, the 
display 50, and the user interface 52. Non-limiting examples of the sensor 44 are 
multispectral sensors, digital cameras, electro-optic framing cameras, and cameras that 
include frame store devices (i.e., charge coupled device (CCD)), such as that produced by 
Phillips. The sensor 44 includes an optical element 54, such as an electronic shutter, that 

35 causes light scattering. The sensor 44 is typically used in aircraft to take pictures of the 
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ground. Non-limiting examples of the user interface 52 are a keyboard, a mouse, or other 
input device for allowing a user to interact with the processor 46. The sensor 44 generates an 
image frame array (i.e., image or sensor information) and sends the image frame array to the 
processor 46 for correction. The processor 46 adjusts the received digital image frame array 
5 to compensate for previously-determined noise of the sensor 44 and light scattering effects of 
the optical element 54. The processor 46 then generates an image based on the corrected 
image frame array and sends the generated image to the display 50 for presentation. It will be 
appreciated that the components shown in FIGURE 1 can be included within a single device 
or may be remotely located from one another over a network and in separate devices. It will 
10 be appreciated that the components of the system 40 are common off-the-shelf components 
and an explanation of these components is not necessary for understanding of the present 
invention. 

Referring now to FIGURE 2, an exemplary process 100 is performed by the 
processor 46 (FIGURE 1). At a block 110, characteristics of a sensor 44 are determined and 

1 5 images generated by the sensor 44 are enhanced according to the determined characteristics. 
The determination of sensor characteristics and adjustment of the image frames produced by 
the sensor 44 are described in more detail below in FIGURE 3. At a block 118, light 
scattering effects are removed from the enhanced image frame. The removal of light 
scattering effects is illustrated in more detail below in FIGURE 8. At a block 120, the image 

20 frame is normalized. Normalization is described in more detail below in FIGURE 6. At a 
block 122, the processor 46 generates an image based on the normalized image frame with 
the removed light scattering effects and sends the generated image to the display 50 for 
presentation or to the memory 48 for storage. 

Referring now to FIGURE 3, an exemplary process 150 determines characteristics of 

25 the sensor 44 and enhances image frames based on the determined sensor characteristics 
from the block 110 (FIGURE 2). Image frames generated by the sensor 44 are produced at a 
predefined interval and are sized based on the array of sensing elements of the sensor 44. The 
array of sensing elements produces an array of frame unit data (hereinafter frame unit). The 
array of frame unit data is an image frame. Each frame unit includes a radiation value (i.e., 

30 radiance). At a block 154, a stable region of interest is selected from a mask region of an 
array of sensing elements of the sensor 44. Selection of the stable region of interest is shown 
in more detail below in FIGURE 4. At a block 156, the image frames used in normalization 
at the block 120 (FIGURE 2) are enhanced using information from the selected stable region 
of interest. The block 156 includes a block 158 that generates and enhances a set of image 

35 frames produced by the sensor 44 being exposed to a known light quantity. The frames 
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exposed to light are created by having the sensor face a calibration device called an 
integrating sphere, which produces a uniform light source of know intensity. At a block 160, 
a set of image frames produced by the sensor 44 being exposed to dark are generated and 
enhanced. Exposing the sensor 44 to dark includes placing a lens cover over the sensor 44. At 
5 a block 162, an image frame generated by the sensor 44 is enhanced. The enhancement that is 
performed at the blocks 158, 160, and 162 is shown in more detail in FIGURE 5. The results 
of the process 150 (FIGURE 3) are enhanced image frames, which include an enhanced 
desired image frame, enhanced dark exposed frames, and enhanced frames exposed to the 
known light quantity. Exposing the sensor 44 to light and dark and enhancing the light and 

10 dark frames are suitably performed under test conditions possibly by the sensor manufacturer 
before product delivery or could be performed by a user before operation of the sensor 44. 

Referring now to FIGURE 4, an exemplary process 200 selects a stable region of 
interest from the block 154 (FIGURE 3). At a block 206, using an image frame produced by 
the sensor 44 under a known dark condition (i.e., exposed to dark), a column of frame units 

15 of the frame image is selected from a known mask area of the sensor 44. The known mask 
area is suitably an area near an edge of the sensor 44 where sensing elements are permanently 
masked or covered by a structure of the sensor 44. In one non-limiting example, the mask 
area suitably is around 20 frame unit columns wide and around 1,024 frame unit rows high. 
At a block 208, a standard deviation for the frame units in the selected columns is 

20 determined. At a block 210, the rows in the selected column that exhibit a desirable standard 
deviation are selected. In one non-limiting example, the rows selected start at row 100 and 
end at row 900. At a decision block 212, the process 200 returns to the block 206 if it is 
determined that a predefined number of columns have not been selected. If at the decision 
block 212 a predefined number of columns have been selected, at a block 216 a mean and 

25 standard deviation for the selected rows and columns are determined over a set number of 
frames. In one non-limiting example, the number of frames that are used is between 100 and 
1000. At a block 218, columns exhibiting the desirable standard deviation over the set of 
frames are selected. The selections performed in the process 200 are suitably performed by a 
user analyzing the determined standard deviation information to determine the desirable rows 

30 and columns to select. It will be appreciated that the selections performed in the process 200 
may be performed automatically by an application program executed by the processor 46. 

Referring now to FIGURE 5, an exemplary process 250 enhances received image 
frames as shown in the block 156 (FIGURE 3). At a block 252, for each image frame 
produced by the sensor 44, a mean and a standard deviation are determined for the frame 

35 units within the selected region of interest in the masked area. At a block 254, a 
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determination is made for the frame units of a produced image frame within the masked area 
that are below a predefined threshold value. In one non-limiting example, the predefined 
threshold value suitably is around 6 times the determined standard deviation for the frame 
image added to the determined mean (mean + (6 x standard deviation)). At a block 256, the 
5 mean is determined only for the frame units below the threshold value. At a block 258, a 
mean difference value is determined based on the determined mean of each frame unit below 
the threshold value and a predefined offset coefficient. This is performed on each frame 
separately. This allows removal of time varying components from the image. The offset 
coefficient is applied to allow noise variations around the dark zero value. In one non- 
10 limiting example, the determined mean is subtracted from coefficient =150 for determining 
the mean difference value. This coefficient is an electrical offset. The offset is used because 
there exists noise when the sensor is tested under the dark condition. It will be appreciated 
that other coefficient values may be used. At a block 260, the determined mean difference 
value is added to each frame unit in the frame image. The process 250 repeats for every 
15 frame image. The results of the block 260 are returned to the respective blocks 158, 160, and 
162 within the enhanced block 156 (FIGURE 3). 

FIGURE 6 illustrates normalization from the block 120 (FIGURE 2). At a block 280, 
a mean is calculated for each frame unit across all of the enhanced image frames associated 
with the dark condition. The result of the block 280 is identified as B. At a block 282, the 
20 mean is calculated for each frame unit of the enhanced image frames exposed to the known 
light condition. The output of the block 282 is identified as A. The calculated mean B for 
each frame unit is subtracted from the corresponding frame unit in the enhanced image frame 
(i.e. array of frame units of the image frame) C at a function 284. The calculated mean B for 
each frame unit is subtracted from the corresponding frame unit in the calculated mean A at a 
25 function 286, thereby removing the mean dark values from the image frames. At a 
function 288, the enhanced image C - B is divided by the mean image frame based on the 
known light condition as follows: C - B/A - B. At a block 290, an average value for the frame 
units in a predefined center area of the frame image A - B is calculated. The calculated 
average value is identified as E. At a function 289, the result of the function 288 is multiplied 
30 by the calculated average E and then added to a coefficient. A non-limiting example of the 
coefficient that is added is 150. Thus, the results of normalization are as follows: 
(C-B/A-B)xE+150 

FIGURE 7 illustrates an image frame A - B, and an area 298 identifies the frame 
units that are used for the calculation of the average done in the block 290 (FIGURE 6). In 
35 one non-limiting example, the area 298 is 16 frame units by 16 frame units. 
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In one embodiment, the original raw data included within the selected region of 
interest from the received image frame (block 110) is reinserted back into the normalized 
image frame. 

Referring now to FIGURE 8, an exemplary process 300 adjusts light scattering 
5 effects resulting from an optical element 54, such as without limitation an electronic shutter, 
within the sensor 44. The process 300 is performed at the block 118 (FIGURE 2). At a 
block 310, the average values of the frame units within the previously-determined region of 
interest of the enhanced image frame are calculated. At a block 312, the calculated average 
for all of the frame units is subtracted from each of the frame units in the enhanced image 

10 frame. The image frame result of the subtraction at the block 312 is considered band data. At 
a block 314, the band data is separated into subarrays of a predefined area and the centers for 
each of the subarray are determined. At a block 318, all the frame units for each subarray are 
totaled thereby generating a total S s for each subarray. At a block 320, the distance of a frame 
unit from the center of a subarray is determined. At a block 322, a previously calculated 

15 effects value is retrieved from an effects table. The effects table includes light scattering 
values that indicate each frame unit's effect on each frame unit, which have been determined 
by measurement of camera characteristics. A frame unit's effect on other frame units is based 
on light scattering properties of the optical element 54. At a block 326, the retrieved effects 
value is multiplied by the associated subarray total S s , thereby generating an effects 

20 value S e ff. At a decision block 328, the process 300 returns to the block 320 if it is 
determined that not all of the subarrays have been analyzed for a frame unit. If all the 
subarrays for a frame unit have been analyzed, at a block 330 all the effects values S C fr are 
totaled for a particular frame unit. At a block 332, the sum of the S e ff values is subtracted 
from the same frame unit. At a decision block 340, if it is determined that not all of the frame 

25 units have been analyzed with respect to all of the subarrays, at a block 344 the process 300 
moves to the next frame unit and returns to the block 320 to repeat until all of the frame units 
have been analyzed. If it is determined that all the frame units have been analyzed, the 
process 300 is complete. 

FIGURE 9 illustrates some of the processing of the process 300 (FIGURE 8) applied 

30 to an enhanced image frame 450. Distances from a first frame unit 460 to centers of each 
subarray 462 are determined. The determined distance value is suitably compared to 
distances stored in a look-up table that is stored in the memory 48. The look-up table 
includes all possible distances between frame units 460 and a corresponding light scattering 
effects value. A retrieved light scattering effects value is multiplied by the sum of all frame 

35 units within the respective subarray. The product produces a relative light scattering effect of 
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the subarray on the frame unit 460. The products for all of the subarrays are added together 
and then subtracted from the frame unit 460, thereby canceling out the light scattering effects 
of all the subarrays upon the frame unit 460. It will be appreciated that other methods of 
canceling out the light scattering effects of frame units upon other frame units can be used. 
For example, the process 300 described above can be performed for determining the direct 
light scattering effect of every frame unit upon each frame unit. 

The S s and scattering effects are fractional numbers requiring floating point numbers. 

The process for removing light scattering effects may be performed on the enhanced 
image frames that were exposed to light. 

It will be appreciated that a frame image may be corrected in real-time if enough 
processing power is provided. Thus, with real-time image generation streaming images 
(video) are possible. 

While the preferred embodiment of the invention has been illustrated and described, 
as noted above, many changes can be made without departing from the spirit and scope of the 
invention. For example, the steps of the processes described above may be performed in a 
different order and still accomplish the same or similar final results. Accordingly, the scope 
of the invention is not limited by the disclosure of the preferred embodiment. Instead, the 
invention should be determined entirely by reference to the claims that follow. 
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